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ABSTRACT 


In  Command  and  Control,  decisions  require  the  fusion  of 
inputs  from  a  number  of  subordinate  decision-makers  during  the 
situation  assessment  process.  The  tactical  operating 
environment  often  introduces  stress  into  the  team's  decision¬ 
making  process.  The  Office  of  Na^  r^tchnology  in  Arlington, 
VA  has  sponsored  research  under  the  T««.;tical  Decision-Making 
Under  Stress  (TADMUS)  program  to  study  ways  to  minimize  the 
degradation  to  the  teams'  effectiveness  during  these  periods. 
Under  the  TADMUS  Project,  the  Tactical  Adaptation  and 
Coordination  Training  (TACT)  experiment  was  designed  by 
Alphatech,  INC.  to  test  theories  on  individual  and  tea^ 
training  techniques  that  were  hypothesized  to  mitigate  the 
effects  of  stress  during  tactical  situations  of  interest  (high 
and  low  stress  scenarios) . 

In  a  detailed  review  of  the  data  gathered  during  the 
experiment,  it  is  concluded  that  the  training  strategies  were 
indeed  effective  in  significantly  altering  the  subject  teams 
ability  to  perform  under  the  test  conditions.  Additionally, 
there  were  no  conclusive  findings  that  the  level  of  stress,  as 
presented  in  the  experiment,  had  a  significant  effect  on  the 
performance  of  the  teams. 
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I .  INTRODUCTION 


A.  BACKGROUND 

In  the  Navy/Marine  Corps  White  Paper  ...From  the  Sea,  a 
fundamental  change  in  strategy  away  from  a  focus  on  a  global 
threat  to  a  focus  on  regional  challenges  and  opportunities  is 
presented  by  Navy  leadership  as  a  result  of  a  fundamental 
shift  in  the  national  security  policy.  As  Joint  Task  Force 
members.  Naval  Forces  will  be  full  participants  in  the 
principal  elements  of  the  National  Security  Strategy — 
strategic  deterrence  and  defense,  forward  presence,  crisis 
response,  and  reconstitution.  Maritime  forces  are 
particularly  well  suited  for  the  forward  presence  and  crisis 
response  missions  which  would  likely  be  conducted  in  the 
complex  operating  environment  of  the  "littoral"  or  near  the 
coastlines  of  the  world.  In  shifting  away  from  open-ocean 
warfighting  on  the  sea,  the  leadership  has  recognized  that  if 
forces  are  to  be  successful  they  must  be  trained  and  equipped 
to  make  near  instantaneous  correct  decisions.  Commanders  will 
be  making  those  decisions  while  operating  task  forces 
constrained  by  less  room  to  maneuver  and  a  far  shorter 
reaction  time  to  a  wider  range  of  more  lethal  threats. 

In  littoral  regions,  friends,  adversaries,  and  neutrals 
are  all  operating  within  confined  and  congested  water  and 
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airspace.  The  normal  density  of  contacts  encountered  while 
operating  near  land  makes  identification  and  situation 
assessment  extremely  difficult  for  the  forces.  (O'Keefe, 
Kelso,  and  Mundy,  1992}  As  a  result  of  the  probable  need  to 
deploy  forces  into  difficult  operating  environments,  where 
mistakes  can  be  catastrophic,  the  Navy  has  pursued  programs 
designed  to  improve  tactical  decision-making  and  performance 
in  stressful  situations. 

The  Tactical  Decision-Making  Under  Stress  (TADMUS)  program 
was  initiated  in  an  effort  to  better  understand  how  tactical 
decisions  are  made  during  high  stress  periods,  when  commanders 
often  are  faced  with  grave  consequences  as  a  result  of 
incorrect  decisions,  which  are  often  made  based  upon  ambiguous 
data  and  uncertain  information.  Past  programs  concentrated  on 
providing  more  time  for  decision  making  by  speeding  up  the 
processing,  dissemination  and  displaying  of  information. 
(Smith  and  Grossman,  1993)  The  TADMUS  program's  primary 
objective  is  to  apply  recent  developments  in  decision  theory, 
individual  and  team  training,  and  information  display  to  the 
problem  of  enhancing  tactical  decision  quality  under 
conditions  of  stress.  The  following  five  tasks  comprise  the 
program's  objectives: 

1.  Definition  and  measurement  of  critical  decision  tasks. 

2.  Examination  of  stress  effects  on  decision-making. 

3.  Development  of  decision  support  principles. 

4.  Development  of  training  and  simulation  principles. 


5.  Development  of  display  principles. 

The  program  consists  of  two  projects;  both  of  which  are 
sponsored  by  the  Office  of  Naval  Technology,  which  is  located 
in  Arlington,  VA.  Project  RM33T40  is  principally  concerned 
with  the  development  of  decision  support  principles  and 
display  principles  for  decision  support  systems.  Project 
RM33T60  is  principally  concerned  with  development  of  training 
and  simulation  principles  to  counteract  stress.  This  thesis 
reports  on  the  Tactical  Adaptation  and  Coordination  Training 
(TACT)  experiment  which  was  designed  to  test  teamwork, 
coordination,  and  adaptation  training  strategies  in  support  of 
completing  task  objective  four.  (Malecki  and  Collyer,  1992) 

B.  PURPOSE 

1.  Tactical  Decision-Making  In  A  Combat  Environment 

In  recent  history,  catastrophic  incidents  such  as  the 
one  involving  the  USS  Stark  and  later  the  USS  Vincennes  have 
focused  attention  on  the  human  factor  in  decision-making  under 
stressful  conditions.  In  retrospect,  it  is  now  acknowledged 
that  improved  training  and  support  must  be  provided  to  the 
decision-maker  to  aid  him  in  sorting  through  confusing 
situations  where  it  is  often  not  clear  who  the  enemy  is  or 
what  his  intentions  may  be.  (Malecki  and  Collyer,  1990)  The 
TADMUS  project  was  initiated  to  explore  methodologies  to 
improve  the  decision-making  process  in  teams  under  stress. 
The  TACT  experiment  is  one  in  a  continuing  series  of  TADMUS 
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program  experiments.  TACT  was  specifically  designed  to  test 
theories  on  individual  and  team  training  techniques  that  were 
hypothesized  to  increase  team  performance  during  periods  of 
high  stress.  Using  the  Decision-Making  Evaluation  Facility 
for  Tactical  Teams  (DEFTT) ,  information  processing  similar  to 
that  of  an  actual  AEGIS  cruiser  combat  information  center 
(CIC)  environment  was  simulated.  Based  upon  variations  in 
levels  of  training  and  differing  levels  of  stressful 
scenarios,  the  DEFTT  lab  was  used  to  measure  the  effect  of  the 
training  techniques  in  question  and  their  ability  to  improve 
team  performance  versus  selected  measures  of  performance. 
(Malecki  and  Collyer,  1992) 

2 .  Questions 

The  TACT  experiment  was  designed  to  test  the  efficacy 
of  training  techniques  aimed  at  enhancing  teamwork  skills  and 
providing  team  coordination  and  adaptation  strategies.  The 
coordination  and  adaptation  strategies  would  be  tested  to  see 
if  they  helped  the  teams  maintain  their  level  of  performance 
in  the  face  of  stress.  The  particular  research  questions  are 
as  follows: 

1.  If  team  members  are  taught  to  monitor  other  team  members 
and  offer  assistance  during  periods  of  high  stress 
(presence  of  heavy  workload  on  a  team  member)  will  the 
teams'  overall  level  of  performance  improve? 

2.  If  teams  are  given  a  technique  to  achieve  a  shared 
mental  model  of  the  current  tactical  picture,  such  as 
structured  TAO  situation  assessments,  would  their 
performance  level  be  improved? 
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Additional  questions  of  interest  for  this  thesis  are: 

1.  Does  an  analysis  of  the  experiment  provide  insight  into 
the  decision  making  process  that  can  be  used  to  improve  the 
effectiveness  of  the  training? 

2.  What  can  be  done  to  improve  the  TACT  experiment? 

3 .  Approach 

For  the  TACT  experiment,  the  overall  approach  was  to 
identify  appropriate  candidate  adaptation  and  coordination 
strategies;  test  those  strategies  in  a  laboratory  setting;  and 
then  evaluate  the  ef '’ectiveness  of  those  strategies.  After 
two  hours  of  initial  DEFTT  lab  orientation  training,  which 
included  individual  workstation  famili^.rization,  the  teams 
engaged  in  two  practice  scenarios  where  they  were  allowed  to 
interrupt  and  ask  questions.  Prior  to  running  each  new 
scenario,  there  was  a  scenario  prebrief  which  included 
specific  tasking  and  a  current  situational  update  for  the 
upcoming  scenario.  Upon  completion  of  the  scenarios,  test 
subjects  were  given  questionnaires  which  were  used  to  gather 
data  on  their  reactions  to  the  scenarios  in  terms  of  the 
level (s)  of  stress  that  were  present  and  their  perceived  task 
workload.  Once  the  familiarization  runs  were  completed,  each 
team  participated  in  two  combat  scenarios,  one  low  stress  and 
one  high  stress.  Observations  of  the  teams'  performance  were 
recorded  using  audio,  video,  and  written  observer  rating 
forms.  Training  intervention  was  then  conducted,  immediately 
followed  by  two  combat  scenarios.  The  data  obtained  in  the 
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later  two  simulations  was  compared  to  data  obtained  in  the 
first  two  simulations  to  determine  the  effectiveness  of  the 
training.  Minitab,  a  computer  based  statistical  package,  was 
later  used  as  the  prime  analysis  tool  in  an  attempt  to  measure 
significant  differences  in  the  levels  of  observed  performance 
of  the  teams  as  a  result  of  receiving  intervention  training.' 

4.  Anticipated  Results 

It  was  hypothesized  that  the  coordination  and 

adaptation  training  strategies  that  were  tested  in  the  TACT 

experiment  would  be  effective  in  counteracting  the  negative 

effects  of  stress  and  would  thus  improve  team  performance 

during  periods  of  high  stress.  The  overall  premise  was 

well  trained  teams  cope  with  stress  through  internal 
mechanisms  of  decision  strategy  adaptation,  coordination 
strategy  adaptation,  and  structural  reconfiguration,  in  an 
effort  to  keep  performance  at  a  required  level  while 
maintaining  stress  below  an  acceptable  threshold  (Serfaty, 
Entin,  Deckert,  and  Volpe,  1993) . 

The  experiment  was  designed  to  look  at  the  following  specific 

points  and  their  relationship  to  managing  the  effects  of 

stress : 

1.  Training  for  implicit  coordination. 

2.  Training  for  team  adaptation  to  stress. 

3.  Periodic  sharing  of  the  TAO's  assessment  of  the 
situation. 

4.  Enhancement  of  the  team's  structural  flexibility. 


'  Minitab  was  the  computer-based  statistical  analysis  software 
package  of  choice  used  by  the  author. 
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If  the  experiment  is  successful,  then  it  will  have 
demonstrated  that  teams  in  a  laboratory  setting  can  be  trained 
to  recognize  the  presence  of  stress,  reconfigure  their 
structure (s)  and  adjust  their  strategies  in  order  to  maintain 
an  acceptable  level  of  performance  when  operating  in  a 
tactically  challenging  environment. 

a.  Support  for  anticipated  results 

Review  of  the  literature  suggests  that  teams  use 
explicit  coordination  during  periods  of  low  stress  and 
implicit  coordination  during  periods  of  increasing  stress. 
Explicit  coordination  requires  more  communications  as  team 
members  generally  respond  to  requests  for  information  etc. 
from  others  as  they  coordinate  their  efforts.  During  periods 
of  stress,  implicit  coordination  becomes  the  desired  norm. 
This  strategy  requires  team  members  to  become  familiar  with 
others  needs;  to  be  able  to  anticipate  requests  and  respond  or 
provide  support  without  prompting.  Implicit  coordination 
relies  upon  team  members  having  a  shared  mental  model  of  the 
situation.^  The  team  members  are  then  able  to  anticipate  the 
needs  and  desires  of  others  that  are  related  to  managing  the 
crucial  task(s)  of  the  moment  and  they  will  thus  refrain  from 
introducing  noise  to  the  situation  assessment  and  decision¬ 
making  process.  (Serfaty,  Entin,  Deckert  and  Volpe,  1993) 

^  A  detailed  discussion  of  the  importance  of  teams  having  a 
common  shared  mental  model  is  presented  in  Rouse,  Cannon-Bowers  and 
Salas  (1992). 
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During  the  TACT  experiment,  the  TAO  is  prompted  to 
periodically  share  his  assessment  of  the  tactical  picture  in 
order  to  provide  the  appropriate  mental  model  which  is 
required  for  implicit  coordination.  After  the  TAO  provides 
his  assessment,  members  then  only  need  to  communicate  relevant 
information  that  deviates  from  his  assessment. 

The  literature  also  suggests  that  teams  should 
maintain  the  flexibility  to  restructure  when  faced  with 
stress.  The  hypothesis  is  that  with  effective  cross  training 
and  functional  task  familiarization,  superior  teams  will  be 
able  to  reconfigure  their  structure  to  support  members  who  are 
experiencing  task  overload  at  their  workstations.  This 
strategy  should  enable  the  team  to  regulate  the  workload 
imposed  by  the  external  stressors.  (Serfaty,  Entin,  Deckert 
and  Volpe,  1993)  During  the  TACT  experiment,  use  of  this 
technique  was  encouraged  and  thus  subsequently  observed  being 
used. 

C.  EXPERIMENTAL  PARTICIPANTS 

The  TACT  experiment  was  conducted  at  two  sites.  Students 
from  the  Joint  Command  Control  and  Communications  (JC^) 
curriculum  at  the  Naval  Postgraduate  School  (NPS)  in  Monterey, 
CA  and  personnel  from  the  Surface  Warfare  Officers'  School 
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The 


Command  in  Newport,  RI  participated  as  test  subjects.^ 
experiment  was  run  by  contract  personnel  from  Alphatech,  INC. 
and  representatives  from  the  Naval  Training  Systems  Center, 
Orlando,  FL.  At  NFS,  student  participants  were  divided  into 
six  teams  with  five  members  each.  Two  teams  acted  as  the 
control  group,  two  teams  received  TACT  training  and  the 
remaining  two  teams  received,  TACT  and  information  structure 
training.  Each  subject  group  had  five  operator  positions, 
tactical  action  officer  (TAO) ,  electronic  warfare  supervisor 
(EWS) ,  anti-air  warfare  coordinator  (AAWC) ,  tactical 
information  coordinator  (TIC)  and  identification  supervisor 
(IDS) .  The  slate  of  operators  reflected  the  current  suite  of 
key  players  in  a  typical  Navy  Aegis  Combat  Information  Center 
(CIC) .  In  addition  to  serving  as  test  subjects,  additional 
students  assisted  contractor  personnel  in  the  mechanics  of 
running  the  experiment  such  as  providing  role  play  and 
completing  subjective  evaluations  of  team  performance.  At  NFS 
a  third  group  was  responsible  for  reporting  their 
interpretation  of  the  data  gathered  during  the  experiment  as 
a  part  of  their  study  of  the  JC^  curriculum. 


^  This  thesis  documents  those  activities  observed  at  NFS 
Monterey,  CA  where  the  author  was  an  experimental  participant  and 
student  in  the  JC^  curriculum. 
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O.  BZPBRZNEIIT  SCOPS 

The  DEFTT  laboratory  was  used  to  conduct  four  simulations 
(two  of  high  stress  scenarios,  two  of  low  stress  scenarios)  to 
evaluate  the  effects  of  the  adaptation  and  coordination 
training  strategies.  If  the  strategies  prove  effective  in  the 
laboratory  tests,  then  those  promising  techniques  would  be 
further  refined  and  subjected  to  additional  testing  in  at-sea 
experiments.  The  effective  strategies  will  eventually  be 
developed  into  a  series  of  recommended  principles,  guidelines, 
and  methodologies  for  training  tactical  decision-making  teams 
throughout  the  fleet.  (Malecki  and  Collyer,  1990) 


[ 

\ 
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II.  EXPERIMENTAL  DESIGN 


A.  OVERVIEW 

The  TACT  experiment  was  developed  by  Alphatech,  INC.  to 
test  hypotheses  regarding  training  interventions  that  would 
reduce  the  impact  of  stress  on  tactical  decision-making.  This 
four  stage  experiment  used  the  DEFTT  laboratory  to  provide 
realistic  simulations  of  AAW  scenarios.  (Entin,  1994)  In 
order  to  keep  the  scope  of  the  experiment  to  a  manageable 
size,  six  key  positions  were  identified  as  those  most  critical 
to  replicating  the  AAW  decision-making  process  inside  of  a 
ship's  CIC.  The  six  positions  which  are  supported  by  the 
laboratory  are:  commanding  officer  (CO) ,  TAO,  AAWC,  TIC,  IDS, 
and  EHS.^  The  lab  also  provides  all  of  the  necessary  tactical 
cues  required  to  support  situation  assessment  and  decision¬ 
making  while  it  records  the  actions  of  team  members  for  later 
analysis  as  they  play  through  the  presented  scenarios. 
(Malecki  and  Collyer,  1992)  Further  details  describing  the 
setup,  the  hypotheses,  assumptions,  statistical  design, 
measures  and  instrumentation  are  provided  in  the  following 
sections. 


*  The  position  of  Commanding  Officer  was  not  utilized  during 
the  TACT  experiment. 
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B.  SETUP 


The  setup  is  comprised  of  the  following  four  general 
categories:  physical,  test  subjects,  special  equipment, 

schedule  of  trials. 

1 .  Physical 

The  DEFTT  lab  was  designed  to  provide  good  functional 
realism  of  an  operational  AAW  environment.  The  setup  consists 
of  six  IBN^AT  386  personal  computers  with  Aegis  display 
system,  command  and  display  system,  and  electronic  warfare 
supervisor  software  modules.  The  six  personal  computers, 
which  serve  as  operator  workstations,  are  networked  to  a 
Hewlett-Packard  9000/345  experimental  control  station  (ECS) . 
The  ECS  generates  and  controls  experimental  scenarios, 
supports  a  multi-channel  communications  system,  and  runs  a 
Barco  Graphics  Large  Screen  Display.^  An  ideal  (uncluttered) 
link  picture  with  Naval  Tactical  Data  System  (NTDS)  symbology 
is  available  to  all  of  the  operators  via  their  personal 
computer  workstations.  (Malecki  and  Col Iyer,  1992) 
a.  Scenarios 

The  DEFTT  scenarios  are  set  in  the  Arabian  Gulf 
area  of  operations.  This  geographic  region  is  a  particularly 
good  setting  for  potential  "real  world"  incidents  due  to  its 
volatility  and  high  density  of  contacts.  The  scenarios  are 


^  The  system  is  capable  of  time-stamped  recording  of  all 
verbal  communications  among  team  members. 


12 


filled  with  ambiguous  situations  which  are  often  left 
unresolved  for  the  team  players.  The  experiment  was  designed 
to  prohibit  decisions  that  were  made  in  a  prior  scenario  from 
influencing  decisions  made  in  later  scenarios.  This  design 
eliminated  a  need  to  provide  feedback  to  teams  on  the 
consequences  of  their  actions  and  the  actions  of  any  contacts 
of  interest  that  were  perceived  as  threats  to  the  team.  This 
technique  was  seen  as  increasing  the  realism  of  the  scenarios 
since  many  real  world  encounters  end  with  questions  of  true 
intent  left  unresolved.*^  (Hutchins  and  Kowalski,  1993) 
b.  Task  Demands 

The  scenarios  were  developed  to  provide  a  stressful 
environment  for  teams  to  be  observed  working  within.^  They 
were  designed  to  require  completion  of  many  complex  tasks 
within  short  periods  of  time.  Contacts  of  interest  were 
blended  in  with  neutral /unknown  and  friendly  tracks,  thus 
making  the  determination  of  their  identity  and  intent  a  more 
complicated  process.  The  uncertainty  and  ambiguity  makes  the 
required  application  of  the  rules  of  engagement  (ROE)  more 
difficult  for  the  teams.  Even  in  cases  where  engagement 


^  The  primary  purpose  of  the  experiment  is  not  to  judge  teams 
regarding  their  right  or  wrong  decisions,  but  rather  to  examine 
their  ability  to  adapt  and  coordinate  their  actions  when  presented 
with  ambiguous  threatening  scenarios. 

^  For  experimental  purposes,  the  level  of  stress  was 
manipulated  through  the  use  of  two  variables:  environmental 
ambiguity  and  operator  workload.  (Entin,  1994) 
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criteria  are  met,  per  the  correct  application  of  the  ROE,  the 
more  appropriate  decision  might  be  not  to  engage. 
Additionally,  the  workload  on  team  members  is  increased  by 
both  internal  and  external  communications  requirements^  while 
very  little  support  from  off-ship  assets  is  provided  to  aid  in 
clarifying  any  ambiguous  situations.  (Hutchins  and  Duffy, 
1993) 

2.  Test  Subjects 

The  six  test  subject  teams  were  composed  of  five 
members  each.  The  team  members  were  tasked  with  filling  five 
of  the  six  key  decision-making  positions  found  in  a  typical 
Aegis  CIC.  All  of  the  subjects  were  student  members  of  the 
JC^  curriculum  at  NFS.  The  subjects  represented  all  the 
services  and  Included  one  civilian  student  from  the  National 
Security  Agency  (NSA) .  All  teams  had  members  with  varying 
levels  of  military  experience,  but  all  of  the  appointed  TAOs 
had  significant  Navy  CIC  operational  experience.  All  of  the 
officers  were  of  the  0-3  and  0-4  paygrades.  The  subjects  were 
unaware  of  which  type  of  training  they  had  received. 
Additionally,  they  were  instructed  not  to  discuss  the 
experiment  with  the  other  teams  until  all  teams  had  completed 
the  experiment.  This  was  done  to  prevent  any  potential 
compromise  of  the  experiment. 


14 


3 .  8p«cial  Bquipmttiit 

The  primary  equipment  required  for  conducting  the 
experiment  was  described  in  the  physical  setup.  A  VHS 
recorder  was  the  only  additional  piece  of  special  equipment 
that  was  not  previously  mentioned.  The  videotape  recorder  was 
used  to  support  later  analysis  by  recording  scenario  events 
from  the  TAO's  large  screen  display  and  the  associated 
communications  (audio)  that  was  occurring  at  the  time  amongst 
the  team  members. 

4 .  Schedule 

The  experiment  was  divided  into  four  phases  and  was 
conducted  in  the  following  sequence; 

1.  Stage  0  —  basic  instruction  and  training  for  all  teams. 

2.  Stage  1  —  pre- intervention  data  collection. 

3.  Stage  2  —  intervention  training. 

4.  Stage  3  —  post-intervention  data  collection. 
Intervention  training,  stage  2,  consisted  of  one  of  the 
following  three  training  levels: 

1.  CONTROL — No  significant  training  or  other  pertinent 
information  was  provided.  This  served  as  a  baseline  for 
team  comparison. 

2.  TACT — Subjects  were  taught  to  identify  signs  and 
symptoms  of  stress.  Subjects  were  then  taught  several  team 
strategies  to  adapt  to  stress  with  a  focus  on  team 
coordination.  Scripted  video  presentations,  which 
demonstrated  good  and  bad  application  of  the  team  behavior 
strategies,  were  then  viewed.  Subjects  were  then  allowed 
to  practice  what  they  had  learned  on  the  DEFTT  simulator 
using  two  practice  scenarios.  To  complete  the  intervention 
training,  subjects  viewed  a  summary  videotape  on  the 


Inportance/ principles  of  teamwork  presented  by  a  retired 
Rear  Admiral. 

3.  TACT  and  Information  Structure  (TACT  plus) — In  addition 
to  the  TACT  training,  the  TAO  was  prompted  every  three 
minutes  to  provide  a  situation  report  (sitrep)  over  the 
network  so  that  other  workstations  would  understand  his 
assessment  of  the  overall  tactical  picture.  Additionally, 
team  members  were  instructed  on  how  to  interpret  and  best 
utilize  the  sitrep  information. 

At  stage  0  (prior  to  commencing  the  experiment) ,  each  of  the 

teams  were  provided  two  hours  of  training.  This  training 

consisted  of  task  familiarization,  two  slide  briefs,  and  a  30 

minute  proficiency  run  using  the  simulator.  At  stages  1  and 

3,  each  teams'  performance  was  observed  and  evaluated  during 

both  a  high  stress  and  a  low  stress  scenario.  Stage  1  data 

observations  were  then  compared  to  stage  3  data  observations 

for  indications  of  the  effectiveness  of  the  training  conducted 

during  stage  2.  (Entin,  1994) 

C.  HYPOTHESES 

The  primary  purpose  of  the  TACT  experiment  was  to  test  the 
hypothesis  that  training  can  mitigate  the  negative  effects  of 
stress  on  team  performance.  The  experiment  specifically 
focuses  on  looking  at  coordination  and  adaptation  strategies 
and  whether  or  not  they  have  the  desired  effect  which  is 
suggested  by  the  theories  set  forth  in  Chapter  I.  The 
contractor  designed  the  experiment  to  look  at  several 
different  measures;  anyone  of  which  could  be  used  to  support 
accepting  the  premise  that  their  proposed  training  strategies 
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did  indeed  effect  team  performance  during  periods  of  stress. 
(Entin,  1994)  This  thesis  will  limit  its  focus  to  only  two  of 
the  five  measures  that  are  supported  by  the  experiment  and 
used  by  the  contractor.^  The  hypotheses  of  interest  are: 

Hqj:  Training  will  not  effect  overall  team  performance. 

H,|:  Training  will  effect  overall  team  performance. 

Training  will  not  effect  teamwork. 

Hj2*  Training  will  effect  teamwork. 

D.  ASSUMPTIONS 

There  were  four  predominate  assumptions  associated  with 
the  TACT  experiment.  First,  DEFTT  is  a  legitimate  simulation 
of  an  actual  Aegis  CIC  environment  (Entin, 1994)  It  is 
therefore  a  valid  tool  for  testing  the  experimental 
condition (s)  of  interest  (decision-making  in  an  at-sea 
tactical  environment  during  high  and  low  stress  periods) . 
Another  assumption  was  that  after  initial  familiarization 
training,  all  teams  were  near  the  same  level  of  competence  and 
understanding  of  the  functionality/buttonology  of  their 
respective  watchstations.  Therefore,  once  the  testing  began 
any  learning  curve  effect  would  be  negligible.  Thirdly,  the 
observers'  ratings  of  team  performance  were  quantitatively 

^  A  more  complete  listing  of  the  measures  will  be  provided 
later  in  the  paper. 

^  Experts  have  accepted  DEFTT  as  a  valid  simulation  of  the 
higher  level  functional  decision  making  process  that  occurs  within 
an  Aegis  AAW  environment. 
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consistent  throughout  the  course  of  the  experiment.  The  final 
significant  assumption  was  that  the  subjects  were  willing  and 
enthusiastic  participants.  This  assumption  was  necessary 
since  the  subjects  were  not  volunteers. 

B.  STATISTICAL  DBSIOBI  OF  EXPERIMENT 

The  statistical  design  of  this  experiment  was  modeled 
after  Campbell  and  Stanley's  Pretest-Posttest  Control  Group 
Design  4  (Entin,  1994) .  Design  4  (adapted  by  Alpha tech,  INC. 
for  the  TACT  experiment)  uses  random  assignment  of  training 
techniques  to  each  team,  in  that  the  team  conditions  (i.e., 
control,  tact,  or  tact  plus)  and  when  they  would  be  exposed  to 
differing  experimental  variables  (i.e.,  low  stress  or  high 
stress  scenario)  were  assigned  prior  to  the  subjects  being 
assigned  to  the  teams.  This  randomization  would  likely 
further  negate  a  potential  compromise  of  the  experiment  that 
might  take  place  if  team  members  were  to  prematurely  discuss 
their  roles  and  the  training  scenarios  which  they  had 
participated  in.  This  experimental  design  also  supports 
multivariate  analysis  which  is  capable  of  looking  at  more 
than  one  independent  and/or  more  than  one  dependent  variable. 
(Campbell  and  Stanley,  1965)  The  experimental  schedule,  is 
presented  in  the  following  table. 


The  author's  personal  observation  was  one  of  enthusiastic 
subjects  who  appeared  to  put  forth  their  best  efforts  commensurate 
with  their  abilities. 
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Tabl«  I:  EXPERIMENTAL  SCHEDULE 


CONDITION 

TEAM  ID  # 

PRE-TRAINING 

POST¬ 

TRAINING 

N5 

2- 

2  + 

1+ 

1- 

CONTROL 

N2 

1+ 

1- 

2- 

2+ 

N3 

1+ 

1- 

2- 

2+ 

TACT 

N1 

1- 

1+ 

2  + 

2- 

N6 

2- 

2+  1 

1  + 

1- 

TACT  & 

1  STRUCTURE 

N4 

1+ 

1- 

1 

2- 

Table  I  depicts  simulation  assignments  to  the  six  teams 
involved  in  the  experiment.  A  key  for  the  table  is 
provided  below. 


«l-«  => 

scenario 

one. 

low 

stress 

'•1+''  => 

scenario 

one. 

high 

stress 

'•2-''  => 

scenario 

two, 

low 

stress 

••2+”  => 

scenario 

two, 

high 

stress 

F.  MEASURES 

The  TACT  experiment  used  four  methods  of  measuring  team 
performance.  The  measurement  instruments  consisted  of  pre  and 
post-mission  questionnaires,  the  NASA  Task  Load  Index  (TLX) , 
trained  observers  conducting  performance  assessments,  and 
observers  quantitatively  measuring  verbal  communication  rates. 
(Entin,  1994)  Only  that  data  collected  as  subjective  measures 
of  teamwork  and  team  performance  will  be  analyzed  within  this 
paper . 
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1.  Pr«-Ml8slon  Questionnaires 

The  pre-mission  questionnaires  were  administered  to 
subjects  before  and  after  intervention  (stage  2)  training. 
The  questions  were  designed  to  assess  the  perceived  congruence 
among  team  members  of  the  same  mental  model  of  the  tactical 
situation.  (Entin,  1994)  If  the  training  was  successful, 
then  the  after  intervention  training  questionnaires  should 
indicate  an  increased  perception  from  members  that  they  and 
the  team  are  more  likely  to  share  the  same  mental  picture  of 
the  situation  as  a  result  of  applying  the  techniques  suggested 
during  the  stage  2  training. 

2 .  Post-Mission  Questionnaires 

The  post-mission  questionnaires  were  administered  to 
subjects  after  each  trial.  The  questions  were  designed  to 
assess  the  level  of  confidence  within  the  team,  the  amount  of 
cross-monitoring  of  teammates  workload  (increased  awareness  of 
the  presence  of  stress)  and  whether  the  level  of  assistance 
and  anticipation  of  other  members  needs  had  increased.  When 
combined  with  the  pre-mission  questionnaires,  the  post-mission 
questionnaires  formed  the  basis  for  a  subjective  self 
assessment  of  team  performance.  (Entin,  1994) 

3.  Subjective  Workload 

The  NASA  TLX  was  used  to  determine  the  workload  that 
subjects  felt  they  were  exposed  to  during  the  course  of  the 
experiment.  This  rating  procedure  relates  workload  to 


demands  imposed  on  the  subject  and  to  the  interaction  of  the 
subject  with  the  task.  The  TLX  responses  are  combined  to 


produce  overall  weighted  workload  scores.  (Hutchins  and 
Kowalski,  1993)  This  measure  could  then  be  used  to  look  at 
perceived  subject  workload  as  it  relates  to  stress  level  and 
if  the  member's  perceived  workload  was  reduced  as  a  result  of 
being  exposed  to  stage  2  training  strategies.  The  implication 
would  be  that  with  a  reduction  in  the  perceived  workload,  as 
a  result  of  the  intervention  training,  an  acceptable  level  of 
performance  will  be  maintained. 

4.  Teamwork  and  Performance 

The  subject  teams  were  rated  by  two  trained  observers. 
The  observers  graded  teamwork  on  the  basis  of  15  individual 
items  which  were  deemed  to  be  appropriate  indicators  of 
teamwork.  The  combined  items  are  referred  to  as  the  AAW  Team 
Observation  Measure  (ATOM).**  In  addition  to  the  teamwork 
ratings,  the  observers  were  also  evaluating  the  teams' 


**  ATOM  is  a  team  process  measure.  It  was  developed  based  upon 
a  critical  incident  approach  and  refined  during  workshops. 
Dimensions  of  the  AAW  team  process  include:  communication,  team 
orientation,  team  leadership,  monitoring,  backup,  feedback,  and 
coordination.  These  behaviors  are  scored  on  a  scale  from  1  to  7 
for  each  event  in  the  scenario.  Note:  the  team  leadership 
dimension  was  not  evaluated  during  the  TACT  experiment.  (Malecki 
and  Collyer,  1992) 
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performance  on  the  basis  of  the  12  individual  items  which  are 
referred  to  as  the  AAW  Team  Performance  Index  (ATPI).^^ 

5.  Team  communication  and  Coordination 

Observers  used  matrices  to  tally  communications  rates 
during  the  scenarios.  These  quantitative  measures  were  to  be 
used  to  assess  any  reduction  in  the  requirement  for 
information  exchange  as  a  result  of  the  training  intervention. 
(Entin,  1994)  In  theory,  members  should  now  be  more  apt  to 
anticipate  communications  needs  and  therefore  pass  pertinent 
information  before  it  is  requested. 

6.  INSTRUMENTATION 

Samples  of  the  various  measuring  instruments  are  provided 
in  Appendix  A. 

H.  TESTING  AND  PILOT  TRIALS 

A  series  of  preliminary  tests  were  conducted  prior  to  the 
TACT  experiment.  Those  tests  looked  at  the  performance 
measurement  instruments,  the  stress  evaluation  methodology, 
and  the  performance  of  the  DEFTT  laboratory  itself.  An 
initial  test  of  the  performance  measures,  using  instructors  at 
the  Aegis  Training  Unit,  Moorestown,  NJ,  revealed  that  the 
scales  were  useable  and  that  they  appeared  to  be  able  to 

ATPI  is  a  team  outcome  measure.  It  is  scored  on  a  scale 
from  1  to  7,  and  is  based  on  the  frequency  of  effective  and 
ineffective  behaviors  exhibited  by  the  AAW  team.  ATPI  has 
demonstrated  a  sensitivity  for  detecting  performance  differences 
due  to  the  presence  of  stress.  (Malecki  and  Collyer,  1992) 


22 


distinguish  between  levels  of  performance.  A  preliminary 
stress  evaluation  methodology  was  developed,  which  uses  a 
stress  matrix,  to  evaluate  stress  levels  within  the  baseline 
scenarios.  The  matrix  became  a  tool  to  measure  the  number  of 
targets  (workload)  versus  the  information  available  about  the 
target  (ambiguity  level) .  Other  testing  and  pilot  studies 
were  done  on  the  DEFTT  lab  to  ensure  that  it  is  capable  of: 
recording  time-stamped  multiplexed  channel  data;  supporting 
workstation  simulation  at  both  operator  and  command  levels; 
and  supporting  networked  simulation  for  simultaneously  testing 
of  multi-operator  teams.  In  addition,  the  DEFTT  scenarios 
were  reviewed  by  experts  to  evaluate  their  "stressfulness"  and 
to  develop  relevant  background  materials  (i.e.,  geopolitical, 
ROE,  order  of  battle)  to  be  used  during  the  experiment. 
(Malecki  and  Collyer,  1992) 

Other  experiments,  which  have  preceded  TACT,  provided  key 
findings  which  were  further  examined  during  the  TACT 
experiment.  Among  those  prior  experiments  were  the  SAINT 
experiment  and  the  CHIPS  experiment.  The  SAINT  experiment 
studied  the  effects  of  team  leader  feedback  on  the  situation 
assessment  process  in  AAW  teams  (Gough,  1992) ,  while  CHIPS 
attempted  to  assess  the  impact  of  human  cognitive  limitations 
on  team  performance  (Armbruster,  1993) .  The  preceding  TADMUS 
program  experiments  yielded  results  which  supported  TACT 
hypotheses  and/or  required  further  evaluation  during  the  TACT 
experiment. 


III.  DATA  DESCRIPTION 


The  TACT  experiment  collected  both  quantitative  and 
qualitative  data  for  analysis.  Data  collection  instruments 
included  pre  and  post-mission  questionnaires,  the  NASA  TLX 
workload  index,  observer  rating  forms,  and  team  communication 
recording  forms.  The  various  collection  instruments  are 
included  in  Appendix  A.  Additionally,  a  videotape  recording 
of  the  proceedings  was  made  to  support  later  analysis. 

A.  EXAMPLE  OF  RAN  DATA 

Raw  qualitative  data  was  recorded  by  trained  observers 
using  the  Teamwork  and  Performance:  Observer's  Rating  Form. 
For  each  observation,  an  interval  marking  scale  from  one  to 
seven  was  provided.  The  forms  provided  brief  descriptions  of 
the  behaviors  that  should  be  observed  for  the  lowest  ratings 
on  the  scale  and  brief  descriptions  of  the  behaviors  that 
should  be  observed  and  associated  with  the  highest  ratings  on 
the  scale.  Using  the  descriptors  as  guides,  the  observers 
then  rated  each  team's  performance  in  the  area  of  interest. 

The  author  uses  the  data  obtained  from  the  rating  forms  to 
support  all  analysis  for  this  thesis.  A  brief  discussion  of 


The  videotape  equipment  was  used  to  record  the  multi-channel 
communications  (both  internal  and  external)  and  the  TAO's  tactical 
presentation  as  seen  on  the  command  and  decision  geographic 
situation  summary  or  large  screen  display. 
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the  uses  of  the  pre  and  post-misslon  questionnaires,  the  TLX 
workload  index,  and  the  team  communication  recording  forms  are 
presented  in  the  following  text  in  order  to  provide  a  more 
complete  picture  of  the  experiment. 

1.  Pre  and  Post-mission  questionnaires — these  forms 
provided  scales  ranging  from  one  to  seven  for  subjects  to 
annotate  their  feelings  regarding  their  confidence  in  the 
teams'  and  their  own  abilities  to  complete  the  assigned 
tasking  and  to  ascertain  if  the  subjects  felt  they  were 
able  to  assist  others.  These  questionnaires  were  designed 
to  support  testing  of  the  shared  mental  model  theories 
presented  in  chapter  I. 

2.  TLX  workload  index — these  forms  provided  a  scale  for 
test  subjects  to  rate  the  previously  completed  missions  in 
terms  of  the  following  six  areas:  mental  demand,  physical 
demand,  temporal  (time  pressure)  demand,  performance, 
effort,  and  frustration. 

3.  CIC  Team  Communication  Recording  Forms — the  TAO's 
communications  and  those  of  the  team  members  were  recorded 
by  observers  using  simple  tick  marks  to  note  each 
occurrence  in  a  matrix  format.  The  matrix  data  will 
support  analysis  of  communications  rates  and  the  nature  of 
the  communications  in  terms  of  them  being  requests, 
transfers  or  acknowledgements. 

All  of  the  data  collected  and  described  above  will  be  used  by 
Alphatech,  INC.  in  their  analysis.  Additional  data  analysis, 
not  discussed  here,  will  be  conducted  by  the  Naval  Training 
Systems  Center  and  by  the  Naval  Command,  Control  and  Ocean 
Surveillance  Center  (NCCOSC)  RDT&E  Division  as  they  also 
review  the  TACT  experiment. 


B.  DATA  PROBLEMS 

No  data  collection  or  interpretation  problems  were 
encountered  relevant  to  this  analysis. 
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C.  DATA  TABLE  COOING  SCHEME 

Each  observer's  scores  were  recorded  on  the  rating  forms. 
Those  values,  plus  specific  identification  and  experiment 
control  variables  were  transcribed  into  a  data  table  of  34 
columns  of  variables  by  48  rows  of  iterations.  (See  Appendix 
B.)  The  first  seven  columns  consist  of  identification  and 
experiment  control  variables.  Of  the  remaining  27  columns, 
the  first  15  correlate  to  the  teamwork  measures  of  the  rating 
form.  The  next  eight  columns  measure  team  performance  in 
general  and  the  final  four  columns  are  associated  with 
performance  specific  to  the  particular  scenario.  The  coding 
scheme  for  the  first  seven  columns  is  presented  in  Table  II. 
The  remainder  of  the  columnar  values  are  transcribed  directly 
from  Appendix  B. 

D.  DATA  REDUCTION 

The  first  step  in  data  reduction  was  to  sort  the  data 
spreadsheet  by  groupings  which  would  support  analysis  of  the 
hypotheses  in  question.  After  sorting  the  data,  averages  were 
computed  for  performance  and  teamwork  scores  using  the  ATPI 
and  ATOM  measures  discussed  in  Chapter  II.  Row  averages  for 
all  of  the  data  entries  were  then  extracted  along  with  their 
associated  identifiers.  After  reducing  and  arranging  the 
initial  spreadsheet  into  a  more  manageable  format,  further 
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Tabl«  IZ:  DATA  CODING  SCHEME 


subdivisions  of  columns  were  completed  as  required  to  support 
a  detailed  categorical  analysis  of  the  experimental  data.  The 
modified  data  spreadsheet  is  provided  in  Appendix  B. 
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IV 


ANALYSIS 


This  chapter  describes  the  analysis  procedures  used  for 
the  TACT  experiment.  It  sets  forth  the  analysis  plan  and  then 
provides  the  detailed  steps  that  were  conducted  during  the 
process.  In  the  final  section  of  the  chapter,  the  detailed 
results  of  the  experiment  are  presented. 

A.  ANALYSIS  PLAN 

The  analysis  of  experimental  results  was  conducted  in  two 
phases.  First,  the  ATPI  outcome  measures  were  used  to  rate 
team  performance  effectiveness,  then  the  ATOM  process  measures 
were  used  to  evaluate  teamwork  behaviors.  Using  the  data 
collected  from  the  observer's  rating  forms,  the  experiment 
attempts  to  detect  differences  in  team  performance  and 
teamwork  evaluation  scores  where  it  is  expected  that  some 
experimental  treatments  will  be  more  effective  than  others. 
Based  on  the  findings,  the  two  null  hypotheses  presented  in 
Chapter  II  will  be  rejected  if  the  sample  evidence  contradicts 
them  and  provides  strong  support  for  the  alternative 
hypotheses . 

The  statistical  analysis  of  the  test  data  was  completed 
with  the  aid  of  Minitab.  Minitab  generates  a  p-value  that 
indicates  the  probability  of  observing  an  outcome  like  the  one 
actually  observed  or  more  unusual,  under  the  assumption  that 
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the  given  dependent  variable  is  effected  the  same  by  all 
levels  of  the  independent  variables  (when  the  null  hypothesis 
is  true).  Tests  of  significance  at  a  =  0.05  will  be  used  as 
rejection  criteria  for  the  null  hypotheses.  Using  Minltab, 
when  the  p-value  is  less  than  the  critical  value  (a  »  o.05) 
one  can  be  95%  certain  that  any  change  in  the  dependent 
variable  was  caused  by  a  change  in  the  independent  variable 
(treatment  condition),  not  a  random  occurrence . 

B.  METHODOLOGY 

The  recommended  analysis  for  "Design  4"  is  based  on  gain 
score,  using  the  analysis  of  covariance  (ANCOVA)  tests 
(Campbell  and  Stanley,  1965).  In  addition  to  these  tests, 
normality  and  scatter  plots  are  used  to  provide  additional 
insight  during  the  analysis  process.  For  each  hypothesis,  an 
initial  ANCOVA  test  is  run  on  the  data.  The  posttraining 
intervention  test  score  is  used  as  the  dependent  variable  and 
the  pretraining  intervention  test  score  as  the  covariate.  The 
gain  scores  were  then  determined  by  comparing  pretraining  and 
posttraining  scores  amongst  all  test  groups.  Using  the  gain 


The  significance  level  a  represents  the  probability  of 
rejecting  the  null  hypothesis  when  the  null  hypothesis  is  true. 
Using  Minitab,  if  the  p-value  is  less  than  a  the  null  hypothesis  is 
rejected. 
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scores,  a  second  ANCOVA  test  is  then  conducted  with 
pretraining  scores  being  used  as  the  covariate. 

C.  RESULTS 

1.  Hypothesis  One:  Test  of  Perfomanoe  Meastires 

The  ANCOVA  test  indicates  that  the  covariate, 
pretraining  score,  has  a  highly  significant  effect  on  the  post 
performance  score  (p-value  »  0.002).  Additionally,  the  test 
indicates  that  one  of  the  experimental  factors  in  question 
(level  of  training)  does  indeed  apparently  account  for  a 
statistical  difference  in  the  posttraining  performance  scores 
of  the  teams  (p-value  =  0.011).  There  is  no  indication  that 
the  other  independent  variable  (level  of  stress)  significantly 
effects  the  performance  scores  across  the  experimental 
conditions.  The  Minitab  printout  is  provided  in  Table  III. 

Graphical  representations  of  the  pretraining  and 
posttraining  mean  performance  scores  were  examined  for  the 
three  experimental  conditions  at  the  two  levels  of  stress. 
These  two  bar  graphs  when  examined  together  show  that  the 
control  groups  high  and  low  stress  performance  was  essentially 
unchanged  throughout  the  experiment.  At  the  other  two  levels 
of  treatment,  mean  performance  scores  increased  for  both 
stress  conditions.  The  observer  is  left  to  conclude  that  the 


The  covariate  is  an  uncontrolled  experimental  variable  that 
influences  the  response  but  is  itself  unaffected  by  experimental 
factors.  Using  the  ANCOVA  technique,  adjustments  are  made  for  the 
covariates  effect  on  the  dependent  variable. 
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T«bl«  lilt  AMCOVA  FOR  FOSTTRAZMIMO  PBRFORKAMCB  SCORES 


MTB  >  AMCOVA  'postpcrf'  s  cond24  atr«ss24; 
SUBC>  Covariataa  'praparf'. 

Factor  Levels  Values 


cond24  3123 

stress24  212 

Analysis  of  Covariance  for  postperf 


Source 

DF 

AOJ  SS 

MS 

F 

P 

Cover 1 at es 

1 

6.5377 

6.5377 

12.83 

0.002 

cond24 

2 

5.8363 

2.9182 

5.73 

0.011 

stress24 

1 

0.3745 

0.3745 

0.74 

0.402 

Error 

19 

9.6808 

0.5095 

Total 

23 

32.2971 

Covariate 

Coeff  Stdev 

t-value 

P 

preperf 

0. 

5436  0.152 

3.582 

0 

.002 

presence  of  training  increases  team  effectiveness  when 
compared  to  the  performance  score  of  the  control  group.  See 
Figure  1  to  compare  the  pre  and  posttraining  scores  by  the 
three  conditions  and  levels  of  stress. 

Examination  of  the  scatter  plot  of  the  posttraining 
versus  the  pretraining  performance  scores  confirmed  some 
additional  expectations.  From  the  ANCOVA  test,  the  positive 
covarlate  coefficient  predicted  that  low  pretraining 
performance  scores  were  an  indicator  of  low  posttraining 
performance  scores  and  that  a  high  pretraining  score  should  be 
followed  by  a  high  posttraining  score.  For  the  most  part  this 
was  the  case.  The  plot  also  confirmed  (as  expected)  little  or 
no  movement  for  about  half  of  the  control  groups  scores  while 
dramatic  improvement  was  shown  for  two  of  eight  of  the  test 
scenarios  where  groups  received  tact  only  training.  The  plot 
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Pretraining  Performance  by  Condition  and  Stress 


Stress:  1«L(Mv2«High  ExpCOnd 

Condition:  1*Controi2«Tact3sTact->^ 


Posttraining  Performance  by  Condition  and  Stress 


Stress:  1«Low2«High  Expcond 

Condition:  1*Controi  2«Tact3«Tact'^ 


also  shows  that  the  tact  plus  groups  happened  to  be  strong 
performers  prior  to  receiving  the  intervention  training.  See 
Figure  2. 

The  ANCOVA  test  using  the  performance  gain  score  as 
the  dependent  variable  and  the  pretraining  performance  score 
as  the  covariate  also  indicates  that  the  level  of  training  is 
responsible  for  differences  in  performance  scores  (p-value  » 
0.011).  Again  stress  does  not  show  as  a  significant  factor, 
while  the  covariate  is  again  determined  to  be  a  highly 
significant  factor  in  the  analysis.  The  Minitab  printout  is 
provided  in  Table  IV. 

Table  IV:  AMCQVA  FOR  PERFORMAMCE  GAIN  SCORES _ 

NTB  >  AMCOVA  'psrfgain'  s  eoad24  stress24; 

8UBC>  covariates  'preperf'. 

Factor  Levels  Values 
cond24  3  1  2  3 

stress24  212 

Analysis  of  Covariance  for  perfgain 


Source 

DF 

ADJ  SS 

MS 

F 

P 

Covariates 

1 

4.6096 

4.6096 

9.05 

0.007 

cond24 

2 

5.8363 

2.9182 

5.73 

0.011 

stress24 

1 

0.3745 

0.3745 

0.74 

0.402 

Error 

19 

9.6808 

0.5095 

Total 

23 

17.5931 

Covariate 

Coeff  Stdev 

t-value 

P 

preperf 

-0. 

4564  0.152 

-3.008 

0 

.007 

A  plot  of  the  performance  gain  scores  versus 
experimental  conditions  and  stress  is  also  insightful.  See 
Figure  3.  When  the  graph  is  partitioned  to  show  the  effects 
of  high  and  low  stress,  it  indicates  a  relatively  dramatic 
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Figure  2:  Scatter  Plot  of  Posttraining  vs.  Pretraining 

Performance  Scores 
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Condition.  1=Control  2=Tact  3=Tact+ 


Q.  p  in  p 


U!e6jj9cj  p  ue9|/\| 


Figiire  3:  Performance  Gain  Scores  by  Experimental  Conditions 
and  Levels  of  Stress 
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Stress:  1=Low2=High  Expcond 

Condition:  1=Control  2=Tact  3=Tact+ 


increase  in  the  low  stress  performance  for  the  tact  group  and 
a  lesser  increase  in  performance  under  the  high  stress 
condition.  In  the  tact  plus  training  group,  the  performance 
gain  was  high  and  equal  for  both  low  and  high  stress 
conditions.  This  is  one  of  the  few  places  in  the  experiment 
that  it  appears  that  stress  is  a  factor.  The  TAO's  situation 
assessments  may  have  been  valuable  in  maintaining  the 
performance  gain  for  the  tact  plus  group  during  the  high 
stress  scenarios  whereas  the  tact  group's  performance  gain 
under  high  stress  was  not  as  dramatic.  It  stands  to  reason 
that  if  the  team  is  focused  on  the  same  contacts  of  interest 
during  a  period  of  relatively  high  stress  then  their  overall 
performance  might  be  better  at  least  in  terms  of  addressing 
what  the  decision  maker  sees  as  critical. 

The  scatter  plot  of  performance  gain  versus 
pretraining  performance  score  is  also  presented  in  Figure  4. 
The  plot  shows  a  weak  linear  relationship  for  pretraining 
scores  and  their  associated  gains.  Again  the  high  performers 
had  less  to  gain  and  the  control  groups  demonstrated  little  or 
no  gain.  This  is  important  because  it  confirms  little  or  no 
performance  increase  due  to  any  learning  curve  effect. 

Normality  plots  for  the  pretraining,  posttraining,  and 
gain  scores  were  also  included  in  the  analysis  (see  Appendix 
C) .  In  all  cases,  the  data  points  were  reasonably  close  to 
normal.  The  assumption  of  normality  for  the  data  set  was 
required  for  the  above  tests  to  maintain  their  validity. 
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Scatter  Plot  of  Gain  vs  Pretraining  Performance  Scores 


Figure  4 
Scores 


Scatter  Plot  of  Gain  vs.  Pretraining  Performance 
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Condition:  1  =Control  2=T  act  3=T  act+ 


2.  Hypothesis  Two:  Tost  of  Teamwork  Measures 


trr 

i 


The  statistical  analysis  performed  for  this  hypothesis 
Is  identical  to  that  described  for  hypothesis  one  with  one 
exception.  The  parameter  being  evaluated  Is  teamwork 
behav  The  ANCOVA  test  Indicates  that  the  experimental 
condition  (level  of  training)  Is  significant  (p-value  » 
0.008).  Neither  the  covarlate  factor  (pretraining  teamwork 
score)  nor  the  level  of  stress  are  significant.  The  Mlnltab 
printout  Is  provided  In  Table  V. 

Graphical  representations  of  the  pretraining  and 
posttraining  mean  teamwork  behavior  scores  for  the  three 
experimental  conditions  at  the  two  levels  of  stress  are 
presented  In  Figure  5.  These  two  bar  graphs  when  examined 
together  show  that  the  control  groups  exhibit  no  evidence  of 
improvement  In  their  teamwork  behavior  score  throughout  the 
experiment.  In  contrast,  both  of  the  groups  that  received 


Table  V:  ANCOVA  FOR  P0STTRAXNIN6  TEAMWORK  SCORES 


MTB  >  ANCOVA  'posttfflwk'  =  oond24  stress24; 
SUBC>  covariates  'pretmwk' . 

Factor  Levels  Values 
cond24  3123 

stress24  212 

Analysis  of  Covariance  for  posttimk 


Source 

DF 

ADJ  SS 

MS 

F 

P 

Covarlates 

1 

1.2731 

1.2731 

2.07 

0.166 

cond24 

2 

7.8512 

3.9256 

6.39 

0.008 

stress24 

1 

0.4713 

0.4713 

0.77 

0.392 

Error 

19 

11.6799 

0.6147 

Total 

23 

28.1317 

Covarlate 

Coeff  Stdev 

t-value 

P 

pretmwk 

0 

.2605  0.181 

1.439 

0 

.166 
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Pretraining  Teamwork  by  Condition  and  Stress 


Stress;  1sLow2sHigh  ExpCOnd 

Condition:  1*Control  2»Tact3=Tact-»‘ _ 

Posttraining  Teamwork  by  Condition  and  Stress 


Stress;  1sLow2sHigh  Expcond 

Condition;  1«Control2sTact3sTact+ 

Figure  5:  Pretraining  and  Posttraining  Teamwork  Behavior 

Scores 
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training  show  positive  movement.  There  is  no  substantial 
graphical  evidence  that  scores  are  influenced  by  the  level  of 
stress  unless  one  argues  that  the  control  group  shows  a 
substantial  decrease  in  their  score  when  presented  with  a  high 
stress  scenario  while  the  other  groups  perform  essentially  at 
the  same  level  regardless  of  the  stress  factor. 

The  scatter  plot  for  the  posttraining  versus 
pretraining  teamwork  behavior  scores  is  presented  in  Figure  6. 
This  plot  reveals  a  dispersed  grouping  of  control  group  scores 
on  the  low  end  and  a  fairly  tight  cluster  of  tact  plus  scores 
on  the  high  end  of  the  scale.  The  graph  also  shows  a  fair 
linear  relationship  of  increasing  scores  as  the  level  of 
training  increases  for  each  of  the  test  conditions. 

The  ANCOVA  test  using  the  teamwork  gain  score  as  the 
dependent  variable  and  the  pretraining  teamwork  score  as  the 
covariate  also  indicates  that  the  level  of  training  is 
responsible  for  differences  in  teamwork  behavior  scores  (p~ 
value  s  0.008).  Again  stress  does  not  appear  to  be  a 
significant  factor,  while  the  covariate  (p-value  »  0.001)  is 
a  factor.  The  Minitab  printout  is  provided  in  Table  VI. 

The  bar  graph  representation  of  the  teamwork  gain 
scores  confirms  that  the  training  did  have  a  relatively 
significant  effect  versus  the  control  group  scores.  However, 
across  the  levels  of  stress,  there  appears  to  be  no  difference 
in  the  relative  gains.  The  bar  graph  is  presented  in  Figure 
7. 
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Tabltt  VZt  AHCOVA  FOR  TEAMirORX  QAIll  SCORES 


MTB  >  ANCOVA  'tBVkgaln'  s  eond24  8tr«ss24; 
SUBC>  Covariatas  'pratmwk' . 

Factor  Levels  Values 
cond24  3123 

stress24  212 

Analysis  of  Covariance  for  tmwkgain 


Source 

DF 

ADJ  SS 

MS 

F 

P 

Covariates 

1 

10.2566 

10.2566 

16.68 

0.001 

cond24 

2 

7.8512 

3.9256 

6.39 

0.008 

stress24 

1 

0.4713 

0.4713 

0.77 

0.392 

Error 

19 

11.6799 

0.6147 

Total 

23 

24.5014 

In  the  next  graphical  analysis,  a  scatter  plot  of  gain 
versus  pretraining  teamwork  behavior  score  is  presented  in 
Figure  8.  The  plot  shows  a  negative  linear  relationship  for 
pretraining  scores  and  their  associated  gains.  Again  the  high 
pretraining  performance  means  less  room  for  gain.  As  with  the 
performance  measures,  normality  plots  for  the  pretraining, 
posttraining,  and  gain  scores  were  conducted.  The  data  points 
were  determined  to  be  reasonably  close  to  normal.  As  with  the 
performance  measures  of  the  first  hypothesis,  the  assumption 
of  normality  for  the  data  was  required  for  the  ANCOVA  tests  to 
maintain  their  validity.  The  normality  plots  are  provided  in 
Appendix  C. 

3.  An  Additional  Observation 

The  previous  data  analysis  focused  on  performance  and 
teamwork  observation  scores  as  dependent  variables  with 
experimental  condition  and  level  of  stress  being  the 
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Figiure  6:  Scatter  Plot  of  Posttraining  vs.  Pretraining 

Teamwork  Behavior  Scores 
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Stress;  1=Low2=Hlgh  PrstTHWk 

Condition:  1=Control  2=Tact  3=Tact+ 


Teamwork  Gain  by  Condition  and  Stress 


Figure  7:  Teamwork  Behavior  Gain  Scores  by  Experimental 


Conditions  and  Levels  of  Stress 
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Condition:  1=Control  2=Tact  3=Tact+ 


s  Pretraining  Teamwork  Scores 


Figure  8:  Scatter  Plot  of  Gam  vs. 

Behavior  Scores 


Pretraining  Teamwork 
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Condition:  1=Control  2*Tact  3=Tact+ 


independent  variables.  During  the  course  of  the  analysis,  the 
data  was  also  checked  for  differences  in  team  scores  between 
the  two  sites.  If  significant  differences  in  site  scores  were 
noted,  then  concerns  would  be  raised  regarding  the  validity  of 
the  experimental  results  due  to  possible  biases  being 
introduced  as  a  result  of  two  different  sets  of  observers 
being  used  between  the  two  test  sites. 

The  following  group  of  illustrations  (Figures  9-14) 
graphically  show  the  dispersions  in  scoring  patterns  between 
the  sites.  In  scatter  plots,  the  NFS  subjects  tended  to  be 
homogeneously  dispersed,  while  the  SWOS  subjects  were 
interspersed  toward  either  end  of  the  scoring  spectrum 
(control  and  tact  groups  rated  near  the  low  end  of  the  scale; 
tact  plus  groups  rated  near  the  high  end  of  the  scale  both 
before  and  after  training) .  Though  the  scoring  between  the 
sites  was  noted  as  peculiar,  it  was  felt  that  the  prior 
analysis  was  not  compromised.'^  When  viewed  separately,  the 
posttraining  scoring  from  SWOS  seems  to  show  more  dramatic 
effects  of  the  training  intervention,  across  the  levels.  On 


To  preclude  against  biased  observations,  it  is  suggested 
that  the  same  observers  be  used  for  all  evaluations  and  that  they 
be  kept  ignorant  as  to  which  test  subjects  received  which 
treatments  (Campbell  and  Stanley,  1965) . 

In  reviewing  the  data,  it  was  noted  that  quantitatively 
higher  mean  scores  were  given  by  the  NFS  observers.  However,  in 
focusing  on  the  effectiveness  of  the  intervention  training 
techniques,  it  was  judged  that  if  all  scores  were  averaged  together 
then  the  effects  of  the  quantitative  differences  would  be  negated. 
(Entin,  1994) 
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the  other  hand,  the  scoring  at  NPS  showed  more  realistic  near 
equal  evaluations  of  team  abilities  prior  to  the  teams  being 
exposed  to  training.  When  the  pretraining  scores  are  also 
considered  this  difference  is  mostly  mitigated.  NPS  data 
alone  also  showed  improvements  with  increasing  intervention, 
though  less  dramatic.  There  is  no  obvious  explanation  for 
this  apparent  anomaly  in  data  between  the  two  sites." 


"  As  stated  in  Chapter  I,  this  paper  focuses  on  the  experiment 
as  observed  from  NPS.  However,  for  an  increased  sample  size,  the 
author  uses  the  data  obtained  from  both  experimental  test  sites 
during  the  analysis  phase. 


46 


Pretraining  Performance  by  Condition  and  Site 


Site:  1*SWOS2=NPS  ExpCOnd 

Condition;  1=Control2»Tact3=Tact+ 
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Posttraining  Performance  by  Condition  and  Site 
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Figure  9:  Pretraining  and  Posttraining  Performance  Scores 
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Scatter  Plot  of  Post  vs  Pretraining  Performance  Scores 


Site:  i*swos2=NPS  Preperf 

Condition:  IgControl  2=Tact  3=Tact+ 


Scatter  Plot  of  Gain  vs  Pretraining  Performance  Scores 


Site:  i=swos2=NPS  Preperf 

Condition:  laControl  2=Tact  3=Tact+ 

Figure  H:  Scatter  Plots  of  Performance  Scores  by  Sites 
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Pretraining  Teamwork  by  Condition  and  Site 
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Posttraining  Teamwork  by  Condition  and  Site 


Site:  1=SWOS2sNPS  ExpCOnd 

Condition:  1=Control2=Tact3=Tact+ 

Figure  12:  Pretraining  and  Posttraining  Teamwork  Behavior 

Scores 
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in  bv  Condition  and  Site 
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Figure  13:  Teamwork  Behavior  Gain  Scores  by  Experimental 

Conditions  and  Sites 
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Site:  1=SW0S2=NPS 
Condition:  1=Control  2=Tact  3=Tact+ 


Scatter  Plot  of  Post  vs  Pretraining  Teamwork  Scores 
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Scatter  Plot  of  Gain  vs  Pretraining  Teamwork  Scores 
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Figure  14:  Scatter  Plots  of  Teamwork  Behavior  Scores  by  Sites 
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V.  CONCLUSIONS 


The  purpose  of  this  chapter  is  to  discuss  the  experimental 
results  with  regard  to  the  two  hypotheses  presented  in  Chapter 
II  and  to  revisit  the  initial  research  questions. 

A.  HYPOTHESES 

Based  upon  the  detailed  analysis  presented  in  Chapter  IV, 
the  two  null  hypotheses  are  rejected  in  favor  of  the 
alternate  hypotheses.  In  both  cases,  data  gathered  during  the 
course  of  the  experiment  supports  the  premise  that  adaptation 

a 

and  coordination  training  strategies,  when  implemented,  have 
a  measurable  effect  on  a  team's  ability  to  perform  under  the 
test  conditions  presented  during  the  course  of  this  study.  A 
summary  table  (i.e..  Table  VII)  is  presented  to  show  the 
effectiveness  of  the  training  strategies  across  the 
experimental  conditions  for  both  of  the  hypotheses. 

B.  RESEARCH  QUESTIONS 

The  primary  research  questions  from  Chapter  II  were: 

1.  If  team  members  are  taught  to  monitor  other  team  members 
and  offer  assistance  during  periods  of  high  stress 
(presence  of  heavy  workload  on  a  team  member)  will  the 
teams  overall  level  of  performance  improve? 

2.  If  teams  are  given  a  technique  to  achieve  a  shared 
mental  model  of  the  current  tactical  picture,  such  as 
structured  TAO  situation  assessments,  would  their 
performance  level  be  improved? 
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Tabl«  VII:  Pre-Posttraining  Scoring  Summary 


MBASURB 

Score :  Mean 
(Standard  Deviation) 

Control 

Tact 

Tact+ 

Performance 

Pre 

3.5573 

(1.1232) 

3.5615 

(1.1853) 

4.6906 

(0.7109) 

Post 

3.5812 

(1.1139) 

4.2792 

(1.0333) 

5.5240 

(0.3032) 

Performance  Gain 

0.0240 

(0.6037) 

0.7177 

(1.1274) 

0.8333 

(0.6632) 

Teamwork 

Pre 

3.4008 

(1.0535) 

3.5383 

(0.9178) 

4.5608 

(0.8671) 

Post 

3.5317 

(0.9236) 

4.2642 

(0.8819) 

5.4225 

(0.5573) 

Teamwork  Gain 

0.1308 

(0.5922) 

0.7258 

(1.3967) 

0.8617 

(0.9237) 

Question  one  was  answered  in  the  affirmative  within  the  data 
analysis  section  of  the  previous  chapter.  The  experimental 
evidence  strongly  suggests  that  once  teams  are  taught 
adaptation  and  coordination  strategies,  their  overall 
performance  level  and  exhibition  of  teamwork  behavior  skills 
are  improved  upon. 

The  second  question  was  not  fully  explored  during  the  data 
analysis.  The  focus  of  the  thesis  became  determining  whether 
or  not  training  had  an  effect  on  the  teams  versus  no  training. 
Therefore,  the  statistical  tests  were  used  to  measure 


comparisons  to  a  baseline  provided  by  the  control  groups  vice 
a  comparison  at  each  incremental  level  of  training.  However, 
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from  the  scoring  summary  table,  it  appears  that  the  additional 
structured  information  (TAO's  sitrep)  provided  by  the  tact 
plus  training  was  beneficial  to  those  teams.  On  the  surface, 
they  consistently  appear  to  rate  higher  than  the  tact  groups 
during  the  experiment.” 

A  third  research  guest ion  was:  could  the  experiment 
provide  insight  into  the  decision-making  process  during 
stressful  periods  which  could  be  used  to  further  improve  team 
performance  under  the  test  conditions?  During  the  course  of 
the  analysis,  there  was  no  evidence  that  the  level  of  stress 
in  the  scenarios  significantly  effected  the  outcomes.^  Noting 
that  stress  does  not  appear  to  be  a  significant  factor  in  the 
experimental  model,  we  are  left  with  no  clear  answers 
regarding  performance  in  low  versus  high  stress  scenarios  and 
whether  the  training  significantly  impacts  one  training 
condition  more  than  the  other.  We  are  left  to  conclude  that 
no  additional  insight  into  the  decision-making  process  under 


”  Statistical  tests  of  significance  were  not  conducted 
between  the  tact  and  tact  plus  groups  to  confirm  that  the 
differences  in  their  apparent  scores  were  not  simply  a  random 
occurrence . 

”  It  is  the  author's  opinion  that  no  experimental  factors  can 
truly  model  the  stress  felt  on  the  high  seas.  Recognizing  this, 
there  is  probably  no  need  to  try  to  distinguish  between  the  high 
and  low  stress  scenarios  presented  in  this  study.  A  ship's  company 
can  be  just  as  stressed  by  a  single  unknown  uncooperative  aircraft 
that  may  be  flying  a  potentially  threatening  profile  as  by  five 
aircraft  and  a  half  dozen  surface  craft  operating  in  the  near 
vicinity  which  is  causing  a  much  greater  temporal  workload. 
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VI.  RECOKMENDATIONS 

The  purpose  of  this  chapter  is  to  provide  reconunendations 
regarding  the  TACT  experiment  based  upon  the  finding  of  the 
data  analysis  and  personal  observations  of  the  author. 

A.  EXPLORATION  OF  SCORING  DIFFERENCES 

During  the  course  of  the  experiment,  different  observers 
were  used  to  rate  different  teams  from  the  two  test  sites.  As 
a  result  of  looking  at  the  differences  in  scores  from  the  two 
sites;  one  is  left  to  question  if  differences  in  team  scores 
were  truly  attributable  to  the  test  factors  or  were  they 
attributable  to  the  observers.  It  is  recommended  that  the 
contractor  review  the  criteria  for  assignment  of  observers  and 
their  training  to  ensure  consistency  in  the  subjective 
evaluation  of  the  teams.  It  is  recognized  that  any  form  of 
subjective  evaluation  is  just  that  i.e.,  "subjective”  but  it 
is  worth  reviewing  the  procedures  to  ensure  that  results  are 
not  tainted  due  to  biased  or  partially  trained  observers. 

B.  GUARDING  AGAINST  BIASING  THE  DECISION-MAKING  PROCESS 

Teams  that  the  received  the  tact  plus  training  generally 

rated  higher  than  the  other  teams.  If  the  difference  in  their 
rating  is  attributable  to  the  TAO's  sitreps,  then  it  is  also 
imperative  that  an  appropriate  counter  training  be  provided  to 
ensure  that  team  members  are  warned  that  while  the  sitrep  is 
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a  tool  to  share  the  TAO's  view  of  the  tactical  picture,  it  is 
not  meant  to  suppress  input.  Team  members  must  be  alert  to 
guard  against  the  confirmatory  biasing  discussed  in  Gough's 
Thesis  (Gough,  1992) .  Operators  subordinate  to  the  TAO  must 
be  trained  that  they  are  to  speak  out  when  they  disagree  with 
the  TAG'S  analysis  for  they  may  be  holding  a  critical  piece  of 
information  that  the  TAO  may  not  be  aware  of  in  formulating 
his  situational  assessment. 
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I  APPENDIX  A.  EXPERIMENTAL  MEASURING  INSTRUMENTS 


TACT  EXPERIMENT  (TADMUS) 

CIC  TEAM  PRE-MISSION  QUESTIONNARE 
TEAM# _  SITE  DATC_  TEAM  POSITION  SCEN# _ 


1 .  How  fwiir**  confidence  do  you  place  in  the  ability  of  the  other  members  of  your  team  to 
accompli^  this  mission? 

Very  Little  l  2  3  4  S  6  7  A  Great  Deal 

2.  Hiow  much  confidence  do  you  dunk  the  other  team  members  place  in  your  ability  to  accomplish 
this  mission? 

Very  Little  1  2  3  4  S  6  7  AGieatDeal 

3.  To  what  extent  should  team  members  be  aware  of  other  team  members  woridoad? 

Very  Little  1  2  3  4  S  6  7  AGieatDeal 

4.  To  what  extent  do  highly  competent  team  members  experience  stress? 

VeryUttle  1  2  3  4  5  6  7  AGieatDeal 

5.  A  team  member's  decision  making  ability  is  as  good  in  stressful  situations  as  it  is  in  non- 
stressfiil  conditions. 

Strongly  Disagree  1  2  3  4  5  6  7  Slroiigly  Agree 

6.  MoniKxing  the  TAO's  performance  for  possible  mistakes  and  errors  tends  to  reduce  the  TAO's 
stature  and  authority. 

Strongly  Disagree  1  2  3  4  5  6  7  Strongly  Agree 

7 .  A  team  member  should  offer  task  help  to  another  team  member  only  if  he/she  is  sure  the  team 
member  needs  it 

Strongly  Disagree  1  2  3  4  5  6  7  Sffongly  Agree 

8.  To  what  extent  should  team  members  monitor  other  team  members  for  signs  of  stress? 

VeryLiule  1  2  3  4  5  6  7  AGieatDeal 

9 .  To  what  extent  should  team  members  mendcm/share  their  own  feelings  of  stress/workload  with 
other  team  members  during  a  mission? 

VeryUttle  1  2  3  4  5  6  7  AGieatDeal 
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10.  Even  when  stressed,  I  peifcmn  effectively  during  critical  aspects  of  the  mission. 

Strangly  Disagree  12  3  4  5  6  7  Strongly  Agree 

1 1.  To  what  extent  should  the  team  members  change  their  work  strategy  in  response  to  high 
stress/workload? 

Very  Little  1  2  3  4  S  6  7  AGieatDeal 

12.  Communications  among  team  members  are  rarely  affected  by  high  stress/high  workload. 

Strongly  Disagree  1  2  3  4  S  6  7  Strongiy  Agree 

13.  To  what  extent  should  team  members  take  account  of  other  team  members  personalities  for 
effective  crew  cooxlination? 

VeryUttle  I  2  3  4  S  6  7  AGieatDeal 

14.  To  what  extent  can  the  efifecdveness  of  crew  coordination  be  lowered  stress/woikload? 

VeryUttle  1  2  3  4  S  6  7  AGieatDeal 

15.  It  is  not  a  good  idea  to  point  out  an  error  committed  by  a  team  member  during  a  mission. 

Strongly  Disagree  1  2  3  4  S  6  7  Strongly  Agree 

16.  To  what  extent  are  reprimands  more  effective  than  discussions  in  eliminating  some  elements  of 
a  team  member's  poor  task  performance? 

VeryUttle  1  2  3  4  S  6  7  AGreatDeal 

17.  To  what  extent  is  understanding  the  CO's/TAO's  concepts/beliefs  of  the  situation/mission 
important  to  a  team  membei's  execution  of  the  mission? 

Very  Little  1234S67A  Great  Deal 

18.  Task  overload  usually  occurs  because  a  team  member  is  not  very  competent 

Strongly  Disagree  1  2  3  4  5  6  7  Strongly  Agree 

19.  Each  team  member  should  watch  for  situations  in  which  external  events  hiiKier  the  performance 
of  other  team  members. 

Strongly  Disagree  1  2  3  4  5  6  7  Strongly  Agree 

20.  Team  members  should  be  able  to  anticipate  each  other’s  information  needs  during  the  mission. 

Strongly  Disagree  1  2  3  4  5  6  7  Strongly  Agree 

1/24/94 
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TACT  EXPERIMENT  (TADMUS) 

CIC  TEAM  POST-MISSION  QUESTIONNAIRE 


TEAM# 


SITE. 


DATE. 


TEAM  POSITION. 


SCEN#_ 


1 .  How  much  confidence  did  you  have  during  the  mission  that  the  TAO  would  successfully 
con^lete  the  mission  ? 


‘  I  ^  I  ’  I  -  I  ^  I  * 


j_lJ 


Very  f-ittlc 


ModeniB 


AGreatDeal 


2.  How  much  confidence  did  you  have  during  the  mission  that  the  ocher  team  members  would 
successfully  complete  the  mission? 


I  ’  I  *  I  ^  I  ‘  I  ^ 


Very  Little  Modenie  A  Gieat  Deal 

3 .  How  much  assistance  did  you  provide  to  other  team  members  as  the  mission  unfolded? 


‘  I  ^  I  ^  I  ^  I  »  I  ‘  I  ’ 


Very  Little 


Modeote 


AGreatDeal 


4.  To  what  extent  did  you  cross-monito:  the  actions  of  other  team  member  as  the  mission 
unfolded? 

'  I  ^  I  ^  I  *  I  ^  I  ‘  I  H 


Very  Little  Moderate  A  Great  Deal 

S .  To  what  extent  were  you  able  to  andcipam  (le.,  predict)  the  actions  and  decisions  of  the  TAO? 


I  '  I  ^  I  ’  I  “ 


’  I  ‘ 


I  ’  I 


Rarely 


Half  The  Time 


All  The  Time 


6 .  To  what  extent  were  you  able  to  anticipate  (i.e.,  predict)  the  actions  and  decisions  of  the  other 
team  members? 


'  I  ^  I  ’  I  *  I  ’  I  ‘  I  ’ 


Rarely 


Half  The  Time 


All  The  Time 
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7a.  What  was  the  most  critical  episode  of  this  mission? 


b.  During  this  episode  to  what  extent  were  you  thinking  and  acting  “in  sync”  with  the  TAO? 


Very  Little  Moderate  A  Great  Deal 

c.  How  do  you  know  that?  _ _ 


d.  During  this  episode  to  what  extent  were  you  thinking  and  acting  “in  sync”  with  other  team 
members? 


Very  Little  Moderate  A  Great  Deal 

e.  How  do  you  know  that?  _ 


Put  an  “X"  on  each  of  the  six  scales  below,  at  the  point  that  matches  best  your 
workload  experience  for  the  mission  you  have  just  accomplished. 


Mental  Demand 


Physical  Demand 


I  i  I 


Performance 


Effort 


Frustration 


Very  Low 

Very  High 

I  I  I  I  I  I  I  1  I  1 

1  1  ■  1  1  1  1  1  . 

1 

Very  Low 

Very  High 

1  .  1  1  1  1  1  1  1  1 

1  1  1  1  1  1  1  1  1 

1 

Very  Low 

VeryHi^ 

1  1  1  1  1  1  1  i  1  1 

1  1  1  1  .  1  ■  1  1  1 

Perfect 

Failure 

1  1  1  1  1  1  1  !  1  I  1 

1  1  1  1  1  1  1  1  1  1 

Very  Low 

Very  High 

1  1  M  ■  1  .1  1  1 

.  1  1  1  ■  M  1  1 

Very  Low 


Very  Hi^ 
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TACT  EXPERIMENT  (TADMUS) 

TEAMWORK  AND  PERFORMANCE:  OBSERVER’S  RATING  FORM 

TEAM# _  SITE _  DATE  OBSERVER _  SCEN# _ 

Instructions  for  Teamwork  Ratings 

Circle  a  number  on  the  scale  accompanying  the  questions  on  the  following  pages  so  that  it  best 
describes  the  behavior  of  the  team  you  just  observed  Consider  each  team  separately.  Try  not  to 
compare  one  team  to  another.  Instead  strive  to  rate  the  behavior  of  a  team  on  an  absolute  scale.  To 
help  you  perform  this  absolute  rating  a  brief  description  of  the  behavior  you  should  observe  for  the 
highest  rating  on  the  scale  and  a  brief  description  of  the  behavior  you  should  observe  for  the  lowest 
rating  on  the  scale  are  provided  for  each  quesdon.  Read  these  guides  or  anchors  carefully  and  refer 
to  them  as  you  rate  the  team  on  each  item.  Feel  free  to  write  comments  or  explanations  for  any 
question. 

The  rating  scales  or  questions  for  teamwork  are  organized  into  six  areas.  To  further  help  you 
in  your  ratings  each  area  is  defined  below.  Please  read  these  definitions  carefully. 


Team  Orientation 

Team  orientation  refers  to  the  commitment  team  members  have  and  exhibit  to  working  together. 
It  implies  that  they  place  the  goals  and  interest  of  the  teani  ahead  of  their  personal  goals.  It  also 
refers  to  the  trust  each  team  member  has  in  the  other  team  members,  team  pride,  and  esprit  de  corps. 

Communication  Behavior 

Communication  involves  the  exchange  of  information  between  two  or  more  team  members  in 
the  prescribed  manner  and  by  using  proper  terminology.  Often  the  purpose  of  communication  is  to 
clarify  or  acknowledge  the  receipt  of  information. 

Monitoring  Behavior 

Monitoring  refers  to  observing  the  activities  and  performance  of  other  team  members.  It 
implies  that  team  members  are  in^vidually  competent  and  that  they  may  subsequently  provide 
fe^back  and  backup  behavior. 

Feedback  Behavior 

Feedback  involves  the  giving,  seeking,  and  receiving  of  information  among  members.  Giving 
feedback  refers  to  providing  information  regarding  other  member's  performance.  Seeking 
feedback  refers  to  requesting  input  or  guidance  regarding  performance.  Receiving  feedback  refers 
to  accepting  positive  and  negative  information  regarding  i^ormance. 

Back'up  Behavior 

Backup  behavior  involves  assisting  the  performance  of  other  team  members.  This  implies  that 
team  members  have  an  understanding  of  other  member’s  tasks.  It  also  implies  that  membus  are 
willing  to  give  and  seek  assistance. 

Coordination  Behavior 

Coordination  refers  to  team  members'  executing  their  activities  in  a  timely  and  integrated 
manner.  It  implies  that  the  performance  of  some  team  members  influence  the  performance  of  other 
team  members.  This  may  involve  an  exchange  of  information  that  subsequently  influences  another 
member's  performance. 


scenarios  1-/1+ 
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Team  Orientation 

1 .  To  what  extent  was  this  team  oriented  toward  teamwork? 


7  Good  team  orientation  could  be  infened  in  a  situation  where  a  team  member  places  the  goals  and  interests  of 
the  team  ahead  of  personal  goals.  Also  may  be  evident  through  the  display  of  trust,  team  pride,  and  esprit  de 
corps,  and  an  awareness  that  teamwork  is  important 

1  Poor  team  orientation  manifests  itself  when  members  pi'  mal  concerns  above  the  team's  success 

(e.g.,  disregarding  or  refusing  to  follow  procedures;  argument  .  and  open  resentment;  and  becoming 
upset  with  a  member's  performance  and  either  i^oring  or  harassing  that  member  are  evidences  of  poor  team 
orientation). 

2 .  To  what  extent  were  errors  caused  by  inadequate  team  communicadon? 


7  Communication  within  the  team  was  always  effective  and  never  responsible  for  errors  or  degraded 
performance. 

1  Communication  was  wholly  inadequate  and  resulted  in  most  of  the  errors  made  by  the  team. 

3 .  To  what  extent  were  errors  caused  by  improper  individual  actions  or  decisions? 

I  ^  I  ^  I  ^  ^  ^  I  ^  t  ^  I 

7  No  acdons  or  decisions  of  a  single  team  member  resulted  in  errors  or  poor  team  performance. 

1  The  xtions  and/or  decisions  by  a  single  team  member  very  frequently  resulted  in  errors  or  poor  team 
performance. 

Comments: _ 

Communication  Behavior 

4 ,  How  well  did  team  members  communicate? 


7  Good  communication  occurs  when  team  members  pass  on  all  important  information  and  clarify  intentions 
and  planned  procedures;  members  obtain  necessary  information  and  acknowledge  and  npcat  messages  to  ensure 
correctness;  members  ensure  that  their  messages  are  received  as  intended. 

1  Poor  communication  occurs  when  team  members  fail  to  pass  on  information  or  intentions,  or  pass  on 
incomplete  communications;  members  fail  to  clarify  information;  members  fail  to  acknowledge  other  member's 
requests  or  reports;  members  disregard  proper  security  procedures  for  communication;  members  use  imjxoper 
terminology;  members  tie  up  the  net  with  irrelevant  communications. 


scenarios  1-/1+ 
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5 .  To  what  extent  did  the  TAO  provide  tacdcal  direction  or  relevant  information  to  other  team 
members,  without  the  other  team  members  having  to  ask  for  it? 


^  1  ^  ^  I  ^  I  I  ^ 

7  TAO  always  provided  imponant  direction  or  information  to  other  team  members  without  being  asked. 

1  TAO  never  provided  direction  or  information  to  other  team  members  unless  specifically  asked. 

6 .  To  what  extent  did  other  team  members  provide  relevant  tactical  information  to  the  TAO, 
without  the  TAO  having  to  ask  for  it? 


7  Other  team  members  always  provided  important  Information  to  the  TAO  without  being  asked. 
1  Other  team  members  never  provided  information  to  the  TAO  unless  specifically  asked. 

Comments: _ 

Monitoring  Behavior 

7 .  To  what  extent  did  team  members  monitor  each  other’s  behavior? 


7  Good  monitoring  occurs  when  team  members  consistently  observe  the  performance  of  the  others  to  ensure 
the  efficiency  of  the  team;  members  nonce  and  are  concerned  with  the  performance  of  the  entire  team;  one 
member  recognizes  when  other  team  members  perform  correctly;  members  consistently  keep  track  of  other 
team  members'  performance. 

1  Poor  monitoring  occurs  when  team  members  fail  to  notice  other  team  members'  performance  on  almost  ail 
occasions;  members  rarely  notice  when  other  team  members  perform  correctly  or  make  a  mistake. 

8 .  To  what  extent  did  team  members  alert  each  other  to  impending  decisions  and  actions? 


7  Team  members  always  alerted  each  other  to  impending  decisions  and  actions;  supporting  information  was 
actively  solicited  from  other  team  members. 

1  Team  members  did  not  keep  each  other  informed  of  impending  decisions  and  actions;  compromises  to 
mission  safety  or  mission  effectiveness  arose  when  a  team  member  waited  for  the  other  to  volunteer  significant 
information. 

Comments: _  _ 
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Feedback  Behavior 

9.  To  what  extent  did  team  members  provide  feedback  to  one  another? 

I  ■  I  »  I  »  I  *  I  ^  I  »  I  H 

7  Good  feedback  behavior  occurs  when  team  members  go  over  procedures  with  orte  aruMher  by  identifying 
mistakes  and  how  to  correct  them:  members  ask  for  input  regarding  mistakes  and  what  needs  to  be  worted  on; 
members  are  corrected  for  mistakes  and  incorporaie  the  suggestions  in  their  procedures. 

1  Poor  feedback  behavior  occurs  when  one  or  more  team  members  makes  sarcastic  comments  to  one  or  more 
members  when  the  scenario  doesn't  go  as  planned;  members  resist  asking  for  advice  and  make  guesses  on 
proper  procedures:  members  reject  time-saving  suggestions  offered  by  other  team  members. 

Comments; _ 

Backup  Behavior 

10.  To  what  extent  did  team  members  provided  backup  to  one  another? 


7  Good  backup  behavior  occurs  when  one  team  member  is  having  difficulty,  makes  a  mistake,  or  is  unable  to 
perform  duties,  and  one  or  more  members  steps  in  to  help,  ensuring  that  the  activity  is  complet^  properly:  one 
or  mote  members  provide  critical  assistance  without  neglecting  their  own  assigned  dudes;  the  member  having 
difficulty  or  overburdened  displays  a  willingness  to  seek  assistance  rather  than  struggle  and  make  a  mistake. 

1  Poor  backup  behavior  occurs  when  one  or  more  members  fail  to  provide  assistance  to  another  member  who 
is  having  difficulty,  makes  a  mistake,  or  is  unable  to  perform  his  duties;  while  providing  assistance,  the 
memben  tends  to  neglect  their  own  dudes:  members  are  unwilling  to  ask  for  help  even  when  it  is  available; 
one  member  provides  needed  assistance,  but  does  not  inform  others  that  he  is  occupied  assisting  another  or 
what  he  has  done;  one  member  displays  an  unwillingness  to  help  others  even  when  asked. 

1 1 .  To  what  extent  did  the  TAO  andcipate  the  need  to  provide  (some)  assistance  to  one  or  more 
team  members? 


7  TAG  consistently  anticipated  the  need  to  provide  assistance  to  other  team  members  during  critical  phases  of 
the  mission. 

1  TAO  never  anticipated  the  need  to  provide  assistance  to  other  team  members  during  criUcal  phases  of  the 
mission;  the  other  team  members  always  had  to  ask. 

12.  To  what  extend  did  the  other  team  members  anticipate  the  need  to  provide  assistance  to  the 
TAO? 


7  Other  team  members  consistently  anticipated  the  need  to  provide  assistatxre  to  the  TAO  during  critical 
phases  of  the  mission. 

1  Other  team  members  never  anticipated  the  need  to  provide  assistance  to  the  TAO  during  critical  phases  of 
the  mission;  the  TAO  always  had  to  ask. 
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13.  Did  the  team  members  adjust  individual  task  itsponsibilides  to  prevent  overload? 

I  *  I  ^  I  ^  I  •*  I  ^  I  ‘  I  ’  I 

7  Team  membeis  were  consistently  aware  of  each  othei's  workload  buildup  and  reacted  quickly  to  adjust 
division  of  task  responsibilities  to  redistribute  workload . 

1  Team  members  were  generally  unaware  of  each  other's  workload  buildup:  little  or  no  aaempt  was  made  to 
adjust  the  distribution  of  task  responsibilities  before  significant  compromises  to  mission  safety  or  mission 
effecdveness  occurred. 

Comments: _ _ _ 

Coordination  Behavior 

14.  To  what  extent  was  the  team's  behavior  coordinated? 


i  ^  I  ^  I  ^  1  *  L  Vl-  ^  I  M 

7  Good  coordination  behavior  occurs  when  team  members  consistendy  pass  critical  information  to  the  other 
members,  thereby  enabling  them  to  accomplish  tasks;  members  consistendy  cany  out  tasks  quickly  or  in  a 
timely  manner  enabling  others  to  carry  out  their  tasks  effecdvely.  Team  members  appear  veiy  familiar  with 
the  relevant  parts  of  one  another's  jobs  and  carry  out  individual  tasks  in  a  synchronized  manner. 

1  Poor  coordination  behavior  occurs  when  team  members  consistendy  carry  out  their  tasks  ineffecdveiy, 
leading  to  other  team  members'  failing  at  their  tasks:  members  carry  out  their  tasks  unpredictably,  leading  to 
delays  in  execution  of  cridcal  tasks;  members  neglect  to  pass  on  cridcal  pieces  of  information  to  one  another, 
leading  to  breakdowns  in  team  performance:  team  memb^  carry  out  their  tasks  with  significant  delays  leading 
to  team  errors. 

15.  How  congruent/similar  were  the  TAO’s  and  the  other  team  members’  understanding  of  the 
mission? 


7  TAO  and  other  team  membeis  were  completely  in  agreement  (i.e.,  congruent)  on  goals,  tasks,  and  concepts 
involving  the  nussion. 

1  TAO  and  other  team  members  were  rarely  in  agreement  (i.e..  congruent)  on  goals,  tasks,  and  concepts 
involving  the  mission. 

Comments; _ 
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OVERALL  AAW  TEAM  PERFORMANCE  ASSESSMENT 
Instructions  for  Performance  Ratings 

Please  assess  the  performance  of  the  team  for  the  following  tasks  and/or  activities  using  the 
scales  provided.  Note  that  a  score  of  7  always  denotes  effective  or  superior  performance,  while  a 
score  of  1  always  denotes  ineffective  or  very  pow  performance.  The  anchors  or  descriptors 
associated  with  the  high  and  low  scores  are  what  you  should  expect  to  see  for  very  effective  and 
very  ineffective  team  performances,  respectively.  They  are  provided  as  guidance  for  your  ratings. 

1.  Making  radar  detection  reports.  This  refers  to  the  report  made  by  any  team  member  who 
verbally  describes  the  radar  contact 

Very  Poor  1  3  4  5  6  7  Superior 

7  The  radar  detection  reports  are  always  accurate,  concise,  and  timely.  Proper  format  (e.g..  AFP-1,  NWP-32)  and 
tetmindogy  are  always  used. 

1  Some  radar  detections  are  never  reported.  Mtmy  reports  are  inaccurate  and  late.  Often  proper  format  and 
terminology  are  qqi  used. 

2.  Making  ESM  detection  repons.  This  refers  to  verbal  reports  of  ESM  detections. 

\ 

Very  Poor  1  2  3  4  5  6  7  Superior 

7  The  ESM  detection  reports  are  always  accurate,  concise,  and  timely.  Proper  format  and  terminology  are  always 
used. 

1  Some  ESM  detections  are  never  reported.  Many  reports  are  inaccurate  and  late.  (Xten  proper  format  and 
terminology  are  OQI  used. 

3.  Identification/Correlation  reports.  This  refers  to  verbal  reports  of  the  correlation  and 
identification  of  contacts. 

Very  Poor  1  2  3  4  5  6  7  Superior 

7  The  IDATorrelaiion  reports  are  always  accurate,  concise,  and  timely.  Prt^ter  format  and  terminology  are  always 
used. 

I  Some  ID/Correlaiions  are  never  made  and/or  reported.  Many  reports  contain  errors  antVor  are  late.  Frequently 
improper  format  and  iiKOtrect  terminology  are  usecL 

4.  Assessment  of  contacts'  hostile  intent.  This  is  typically  based  on  input  from  lower  levels 
within  the  team  and  made  by  the  TAO  or  CO. 

Very  Poor  1  2  3  4  5  6  7  Superior 

7  TAO/CO  routinely  assess  the  threat  of  each  new  contact  and  advise  the  rest  of  the  team  accordingly.  Assessment 
is  fumly  based  on  information  the  team  has  collected  (e.g.,  ESM.  ID/Corr,  Intel)  and  on  verbal  discussions 
concerned  with  weapons  loads,  flight  profiles,  and  attempts  at  communication  with  the  contact 

1  TAO/CO  infrequently  assess  the  threat  of  new  contacts  and/or  rarely  advise  the  rest  of  the  team  as  to  the 
contact's  threat.  Assessment  is  often  not  based  on  available  information  and  veibal  discussion  about  such  aspects  as 
weapons  load  and  flight  profile  have  not  occurred. 
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5.  Monitoring  he  threat  This  pertains  primarily  to  critical  contacts  of  interest  (CCOI). 

Very  Poor  1  2  3  4  5  6  7  Superior 

7  CCOIs  are  frequently  hooked  and  observation  of  them  are  more  (M*  less  constant.  The  status  of  the  CCOIs  are 
frequently  discuss^  and  ^praised  •  in  short  the  intensity  of  involvement  with  these  threats  is  high. 

1  CCOIs  are  frequently  neglected  or  overlooked.  The  status  of  CCPIs  are  not  reviewed,  discussed,  or  appraised 
frequently  enough  •  in  short,  the  intensity  of  involvement  with  these  threats  is  tow. 

6.  Taking  appropriate  action  in  accordance  with  ROE.  This  refers  to  whether  the  team  decides  to 
take  some  action  against  a  given  CCOI  vs.  failure  to  do  anything  about  it. 

Very  Poor  1  2  3  4  5  6  7  Superior 

7  TAO  (or  CO)  and  team  consistently  take  effective  and  appropr^  actions  to  deal  with  threats.  This  includes 
assigning  CAP.  covering,  issuing  verbal  warnings,  increasing  readiness/going  to  GQ.  activating  doctrine,  and 
detaining  chaff  solutions. 

1  TAO  (or  CO)  and  team  are  lax  and  often  fail  to  take  effective  or  appropriate  actions  to  deal  with  threats.  They 
tend  to  over  react  or  fails  to  react. 

7.  Planning  for  the  upcoming  mission.  This  refers  to  all  planning  activities  performed  by  the  TAO 
or  other  team  members  for  the  upcoming  mission. 

Very  Poor  1  2  3  4  5  6  7  Superior 

7  The  TAO  and/qr  other  team  members  spend  a  reasonable  amount  of  time  planning  for  the  upcoming  mission. 
Roles  are  further  defined  and  tasks  that  are  outside  normal  responsibility  assigned.  Critical  events  that  might  occur 
ate  Clearly  defined  and  qrecifre  responses  agreed  upon. 

1  The  TAO  andtor  other  team  members  spend  little  or  no  time  planning  for  the  upcoming  mission.  Roles  ate  not 
further  d^ned  and  tasks  that  are  outside  normal  responsibility  are  not  assigned.  Little  or  no  discussion  occurs  about 
critical  events  that  might  occur.  Those  events  that  are  mentioned  are  not  tkfined  well  nor  are  reqionses  to  the  events 

<Vlin«ilwrf 


8.  Overall  performance  ratine  of  this  team  for  this  scenario. 

Very  Poor  1  2  3  4  5  6  7  Superior 

7  Superior  teams  have  consistently  scored  well  on  the  above  six  areas,  as  well  as  on  other  unassessed  areas. 
1  Poor  teams  have  consistently  scored  poorly  on  the  above  six  areas,  as  well  as  on  other  unassessed  areas. 
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9.  Performance  of  critical  events.  Below  are  four  critical  events  that  occurred  in  this  scenario. 
Rate  how  the  team  performed  each  on  the  seven  point  scales  provided. 

a.  Four  Iranian  F4s  detected . 

Very  Poor  1  2  3  4  5  6  7  Superior 

b.  Iranian  bogies  split  into  two  sections. 

Very  Poor  1  2  3  4  5  6  7  Superior 

c.  APQ120  detected  (banian  F4). 

VeryPoor  1  2  3  4  5  6  7  Superior 

d.  Low  F4s  pop-up  at  46nm. 

VeryPoor  1  2  3  4  5  6  7  Superior 
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TACT  EXPERIMENT  (TADMUS) 

CIC  TEAM  COMMUNICATION  RECORDING  FORM:  TAO 


TEAM# 


SITE 


DATE  OBSERVER. 


SCEN#. 


PERIOD  1  2  3 


TYPE  &  CONTENT 


ACTION 

&TASK 


PROBLEM 
SOLVING  & 
PLANNING 


INFOR¬ 

MATION 


ACTION 

ATASk 


PROBLEM 
SOLVING  A 
PLANNING 


ACKNOWLEDGMENTS 


TRANSF 


TAO 
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TACT  EXPERIMENT  (TADMUS) 

CIC  TEAM  COMMUNICATION  RECORDING  FORM:  Team 
TEAM# _  SITE _  DATE  OBSERVER _  SCEN# _  PERIOD  1  2  3 


ItATiOM 


ACTION 

&TASK 


PROBLEM 
LYING  A 
PtJkNNING 


TRANSFER! 


INFOR¬ 

MATION 


ACTION 
AT  ASK 


PROBLEM 
SOLVING  A 
PLANNING 


ACKNOWLEDGMENTS 


Additionai  notes  /  Other  categories: 
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APPENDIX  C.  NORMALITY  PLOTS 


AiniqeqoJd 
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Preperf 

Average:  3.93646  Anderson-Darling  Normality  Test 

Std  Dev:  1 .12344  A-Squared:  0.957 

N  of  data:  24  p-value:  0.013 


Normal  Probability  Plot 

Posttraining  Performance  Data 
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Postperf 

Average:  4.46146  Anderson-Darling  Normality  Test 

Std  Dev:  1 .18500  A-Squared:  1 .069 

N  of  data:  24  p-value:  0.007 
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Std  Dev:  0.874594 
N  of  data:  24 


Normal  Probability  Plot 

Pretraining  Teamwork  Behavior  Data 


AjHiqcqoJd 
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Average;  3.83333  Anderson-Darling  Normality  Test 

Std  Dev;  1 .04994  A-Squared;  0.41 1 

N  of  data;  24  p-value;  0.316 


Normal  Probability  Plot 

Posttraining  Teamwork  Behavior  Data 


/^iliqeqojd 


80 


A  Posttmwk  ^  A.  . .  T  . 

Average:  4.4061 1  Anderson-Oarling  Normality  Test 

Std  Dev;  1 .10595  A-Squared:  0.549 

N  of  data:  24  p-value:  0.141 
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